Method of calibrating a direction of a pan, tilt, zoom, camera with respect to a fixed camera, and a system in which such a calibration is carried out

ABSTRACT

A method of calibrating a pan, tilt, zoom (PTZ) camera with a fixed camera utilizing an overview image of a scene captured by the fixed camera, and an image of the scene captured by the PTZ camera when directed in a first direction. By matching features in the overview image and the PTZ camera image, a first calibration is carried out by correlating the first direction to matching features in the overview image. A mapping between the PTZ camera image and the overview image is defined based on the matching features. The mapping is used to map an object from the PTZ camera image to the overview image. Based on an appearance of the mapped object, a quality of the mapping is calculated. If the quality is not good enough, the PTZ camera is redirected to a second direction, and a further calibration is carried out by again.

FIELD OF INVENTION

The present invention relates to the field of cameras. In particular, itrelates to a method and system for calibrating a direction of a pan,tilt, zoom camera with respect to a fixed camera.

BACKGROUND

There exist camera systems, such as the Axis Q6000-E series, whichintegrate one or more fixed sensors with a movable, pan, tilt, zoom(PTZ) camera. The fixed sensors may typically be arranged to provide anoverview of a scene, such as a complete 360° field of view, whereas thePTZ camera may be arranged to be directed to and zoom in on specificportions of the scene. In such a system, a user may, for instance,indicate a specific portion of the scene towards which the PTZ camerashould be directed by clicking in an overview image provided by one ofthe fixed sensors. Following such a click, the PTZ camera will bedirected to and/or zoom in on the specific portion of the scene.

The viewing direction (i.e., the pan and tilt settings) and/or the zoomof the PTZ camera may thus be controlled by clicking in the imagescaptured by the fixed sensors. Such control of the PTZ camera may relyon a relation between positions in the images captured by the one ormore fixed sensors on the one hand, and directions of the PTZ camera onthe other hand. Typically, the required relation may be determined fromthe geometry and optics of the camera system, including the relativepositions of the fixed sensors and the PTZ camera, once the direction ofthe PTZ camera has been initially calibrated with respect to the fixedsensors. More specifically, as the PTZ camera is installed in the camerasystem, its viewing direction in relation to those of the fixed sensorsis not known, and it therefore needs to be calibrated. Such calibrationtypically aims at finding a correlation between a position in an imagecaptured by one of the fixed sensors and a direction of the PTZ camera.

The patent application published as SG 191452 A1 describes afeature-matching approach for calibrating a PTZ camera with respect to awide field of view camera. In particular, a method for determining acorrelation between a coordinate of an image captured by the wide fieldof view camera and PTZ-values of the PTZ camera is provided. Theprecision of the method of SG 191452 A1 relies on the accuracy of thematching of features in the wide field of view image and in anoverlapping image captured by the PTZ camera image. The accuracy of thefeature matching may in turn depend on several factors, such as thenumber of features being present in the images and properties of thelenses of the cameras. In the former example, it may happen that the PTZcamera is directed towards a portion of the scene where few objects arepresent, thereby resulting in few relevant features in the images. Inthe latter example, it may happen that, due to barrel distortion orother geometric distortions in the wide field of view image, it isdifficult to accurately match the features. In both of these examples,the precision of the calibration may in the end be suffering. There istherefore need for improvements.

SUMMARY

In view of the above an improved precision when calibrating a PTZ camerain relation to a fixed camera is presented.

According to a first aspect, a method of calibrating a direction of apan, tilt, zoom, PTZ, camera is performed with respect to a first,fixed, camera, comprising:

receiving an overview image of a scene captured by a first, fixed,camera,

directing a PTZ camera in a first direction,

when the PTZ camera is in the first direction, performing the steps of:

a) receiving an image of the scene captured by the PTZ camera, wherein afield of view of the image captured by the PTZ camera partly overlaps afield of view the overview image,

b) identifying a first set of features in the image of the scenecaptured by the PTZ camera,

c) localizing the first set of features, or a subset thereof, in theoverview image so as to associate the first set of features in the imagecaptured by the PTZ camera with a second set of features in the overviewimage,

d) logging positional data of the second set of features in the overviewimage,

e) defining a mapping between the image captured by the PTZ camera andthe overview image based on the first set of features and the second setof features, mapping an object in the image captured by the PTZ camerato the overview image by using the defined mapping, and calculating aquality of the mapping based on an appearance of the object aftermapping to the overview image,

performing a first calibration of the PTZ camera by correlating thefirst direction of the PTZ camera with the positional data of the secondset of features being logged when the PTZ camera is directed in thefirst direction,

in case the quality of the mapping is below a first threshold:

redirecting the PTZ camera to a second direction,

performing steps a)-d) when the PTZ camera is in the second direction,and

performing a second calibration of the PTZ camera by correlating thesecond direction of the PTZ camera with positional data of the secondset of features being logged when the PTZ camera is directed in thesecond direction.

According to the above method, a first calibration based on featuremapping is carried out when the PTZ camera is directed in a firstdirection. If it is found that the quality of the mapping is not goodenough, i.e., that the quality is below a threshold, the PTZ camera isredirected to a second direction, and a second calibration based onfeature mapping is carried out when the PTZ camera is in the seconddirection. By re-directing the PTZ camera and repeating the calibrationwhen the quality is not good enough, the quality and precision of theresulting calibration may be improved.

By calibration of a direction of a PTZ camera with respect to a fixedcamera is generally meant to find a correspondence between a directionof the PTZ camera and a position in an image captured by the fixedcamera.

By a mapping between the image captured by the PTZ camera and theoverview image is generally meant a function or transformation whichmaps points in the image captured by the PTZ camera to points in theoverview image. The mapping may, e.g., be defined by a transformationmatrix.

By quality of a mapping is generally meant a metric which is evaluatedbased on an appearance, such as size and shape, of an object after themapping has been applied. The quality of a mapping is typically ameasure of how well the mapping preserves the appearance of an object.

The method may further comprise performing step e) when the PTZ camerais in the second direction, wherein the step of performing a secondcalibration of the PTZ camera is made on a condition that the quality ofthe mapping calculated in step e) when the PTZ camera is in the seconddirection is greater than or equal to the first threshold. In this way,the second calibration is only made if the quality of the mapping whenthe PTZ is in the second direction is good enough.

The method may further comprise keep redirecting the PTZ camera tofurther directions, and repeating steps a)-e) until the quality of themapping calculated in step e) is greater than or equal to the firstthreshold. The PTZ camera may thus be redirected until a mapping of goodenough quality is achieved. In this way, the precision of thecalibration may be further improved.

As further discussed above, the accuracy of a feature matching, and inturn the quality of a mapping defined by the matching set of features,may depend on the number of features present in the images. For example,if the PTZ camera, when directed in the first direction, is directedtowards a region in the scene where few objects are present, there willlikely be few features in the image of the scene captured by the PTZcamera to base the mapping upon. As a result, the accuracy of thematching, and thereby the quality of the mapping, will typically beworse than if more features had been present in the PTZ camera image. Inorder to improve the quality of the mapping, the PTZ camera maytherefore be re-directed to a portion of the scene where more objectsare present, thereby resulting in more features in the image captured bythe PTZ camera. Such a portion of the scene may be located byconsidering the overview image and identifying an area thereincomprising many features. When such an area in the overview image hasbeen identified, the PTZ camera may be redirected, using the firstcalibration as an initial calibration, such that the PTZ camera capturesan image covers or at least overlaps the identified area. In moredetail, the method may comprise: identifying an area in the overviewimage where a density of features in the overview image exceeds a secondthreshold, and selecting the second direction on basis of the firstcalibration of the PTZ camera such that an image captured by the PTZcamera when directed in the second direction covers the identified areain the overview image. The density of features may, e.g., be calculatedas the number of features per unit area in the overview image.

Another factor that may affect the accuracy of the feature matching, andin turn the quality of a mapping defined by the matching set offeatures, is the properties of the lenses of the cameras. For example,the overview image may, in contrast to the image captured by the PTZcamera, be captured by a wide-angle-lens which gives rise tobarrel-distortion or other distortions, such as pincushion distortion ormoustache distortion, of the overview image. These distortions affectperspectives and proportions in the resulting overview image. Suchdistortions will typically be more pronounced at the periphery of theoverview image, and less pronounced in the center of the overview image.Therefore, one may expect that the accuracy of the matching of featuresin the overview image and the image captured by the PTZ camera may behigher if the PTZ camera is directed such that it points to a portion ofthe scene which is depicted at the center of the overview image. Themethod may therefore comprise: selecting the second direction on basisof the first calibration of the PTZ camera such that an image capturedby the PTZ camera when directed in the second direction covers a centerof the overview image. The PTZ camera may thus be redirected, using thefirst calibration as an initial calibration, such that the imagecaptured by the PTZ camera covers or at least overlaps the center of theoverview image.

As mentioned above, the first set of features identified in the imagecaptured by the PTZ camera and the second set of features identified inthe overview image is used to define a mapping, e.g., in the form of atransformation matrix. By that mapping, points in the image captured bythe PTZ camera may be mapped to points in the overview image. Themapping may also be used to map an object in the image captured by thePTZ camera to the overview image. For example, each point of the object,or selected points of the object may be mapped using the definedmapping. The appearance of the object when mapped to the overview imagemay then be used to calculate a quality of the mapping. In particular,calculating the quality of the mapping may include calculating asimilarity between an appearance of the object in the image captured bythe PTZ camera and an appearance of the object after mapping to theoverview image. If the object has a similar appearance after themapping, the quality of the mapping is considered to be high, while ifthe mapping changes the appearance a lot, the quality of the mapping isconsidered to be low. The quality of a mapping is hence a measure of howwell the mapping preserves the appearance of an object.

The method may further compensate for barrel-distortion in the overviewimage when calculating the similarity of the appearance of an objectbefore and after mapping to the overview image. For example, assume thatthe object in the image captured by the PTZ camera has a rectangularshape. Even if a perfect match of features, and a perfect mapping isfound, the rectangular shape would still not be rectangular in theoverview image, since the barrel distortion alters perspectives andproportions of the rectangular shape. It would therefore make more senseto compare the appearance of the rectangle after mapping to the overviewimage to the shape that a rectangle would have when altered by thebarrel distortion, rather than by comparing it to the originalrectangular shape. Therefore, the method may, prior to calculating thesimilarity, adjust the appearance of the object in the image captured bythe PTZ camera on basis of properties of a lens system of the firstcamera. In this way, dissimilarities in appearance of the object causedby barrel-distortion of the overview image will not influence thesimilarity calculation, and thereby not influence the determined qualityof the mapping. The amount of barrel-distortion may be deduced from theproperties of the lens system of the first camera.

Alternatively, barrel-distortion of the overview image may be taken intoaccount by tolerating a lower quality of the mapping closer to theboundaries of the overview image. This may be realized by allowing thefirst threshold, to which the quality of the mapping is compared, todecrease with a distance from the center of the overview image. Morespecifically, the first threshold may depend on the positional data ofthe second set of features, such that the first threshold decreases witha distance from a center of the overview image.

The appearance of the object may be at least one of a size of the objectand a geometrical shape of the object.

The object in the image captured by the PTZ camera may correspond to aperiphery, i.e., the boundary, of the image captured by the PTZ camera.Thus, the object does not need to correspond to an object in the scene.The periphery of the image captured by the PTZ camera may be mapped tothe overview image by mapping the four corner positions, i.e., the fourcorner coordinates of the image captured by the PTZ camera, to theoverview image using the mapping. The object in the image captured bythe PTZ camera may thus have a rectangular shape.

A feature, such as the features of the first set of features and thesecond set of features, may include at least one of an edge or a cornerin a captured image of the scene. A feature may further be associatedwith attributes, such as color, size and/or direction. The attributes ofa feature identified in the image captured by the PTZ camera may be usedwhen localizing a corresponding feature in the overview image. In thisway, the feature matching may be simplified and made more accurate.

The first fixed camera and the PTZ camera may be part of a system whichincludes further fixed cameras. In case the relation between the firstfixed cameras, such as their relative positions and viewing directions,is not known, the above method should be repeated for each fixed camera.If the relation between the fixed cameras is known, the calibrationcarried out with respect to the first fixed camera may be used in thecalibration of the further fixed cameras in the system. In such case,the method may further comprise: capturing a further overview image ofthe scene using a second, fixed, camera having a known position anddirection in relation to the first camera, and calibrating the PTZcamera with respect to the further overview image of the scene capturedby the second camera on basis of the first or second calibration of thefirst camera and the known position and direction of the second camerain relation to the first camera.

According to a second aspect, the above is achieved by a system,comprising:

a first, fixed, camera arranged to capture an overview image of a scene,

a pan, tilt, zoom, PTZ, camera which is separate from the first, fixedcamera, and

a controller operatively connected to the first camera and the PTZcamera, the controller being configured to direct PTZ camera in a firstdirection, and, when the PTZ camera is in the first direction, toperform the steps of:

a) controlling the PTZ camera to capture an image of the scene, whereina field of view of the image captured by the PTZ camera partly overlapsa field of view of the overview image,

b) identifying a first set of features in the image of the scenecaptured by the PTZ camera,

c) localizing the first set of features, or a subset thereof, in theoverview image of the scene so as to associate the first set of featuresin the image captured by the PTZ camera with a second set of features inthe overview image,

d) logging positional data of the second set of features in the overviewimage,

e) defining a mapping between the image captured by the PTZ camera andthe overview image based on the first set of features and the second setof features, mapping an object in the image captured by the PTZ camerato the overview image by using the defined mapping, and calculating aquality of the mapping based on an appearance of the object aftermapping to the overview image,

the controller further being configured to:

perform a first calibration of the PTZ camera by correlating the firstdirection of the PTZ camera with the positional data of the second setof features being logged when the PTZ camera is directed in the firstdirection,

in case the quality of the mapping is below a first threshold:

redirect the PTZ camera to a second direction,

perform steps a)-d) when the PTZ camera is in the second direction, and

perform a second calibration of the PTZ camera by correlating the seconddirection of the PTZ camera with positional data of the second set offeatures being logged when the PTZ camera is directed in the seconddirection.

The system may further comprise:

at least one further fixed camera, wherein the first camera and the atleast one further fixed camera are directed in different directions soas to capture overview images covering different portions of the scene,

wherein the PTZ camera is mounted in relation to the first camera andthe at least one further fixed camera such that the PTZ camera isdirectable to capture images which overlap an overview image captured bythe first camera, and to capture images which overlap an overview imagecaptured by the at least one further fixed camera.

According to a third aspect, there is provided a computer programproduct comprising a (non-transitory) computer-readable medium havingcomputer code instructions stored thereon for carrying out the methodaccording to the first aspect when executed by a processor.

The second and third aspects may generally have the same features andadvantages as the first aspect. It is further noted that the inventionrelates to all possible combinations of features unless explicitlystated otherwise.

BRIEF DESCRIPTION OF THE DRAWINGS

The above, as well as additional objects, features and advantages willbe better understood through the following illustrative and non-limitingdetailed description of embodiments with reference to the appendeddrawings, where the same reference numerals will be used for similarelements, wherein:

FIG. 1 schematically illustrates a camera system according toembodiments.

FIG. 2 illustrates overview images of a scene captured by fixed camerasand an image captured by a movable, PTZ, camera.

FIG. 3 is a flowchart of a method according to embodiments.

FIG. 4 schematically illustrates an overview image captured by a fixedcamera and an image captured by a movable, PTZ, camera.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The description will be set forth more fully hereinafter with referenceto the accompanying drawings, in which embodiments of the invention areshown. However, the teachings may be embodied in many different formsand should not be construed as limited to the embodiments set forthherein; rather, these embodiments are provided for thoroughness andcompleteness, and to fully convey the scope of the invention to theskilled person. The systems and devices disclosed herein will bedescribed during operation.

FIG. 1 illustrates a camera system 100. The camera system 100 may, forexample be used to monitor a scene. The camera system 100 comprises oneor more fixed cameras 102 a, 102 b, a movable camera 104, and acontroller 106. The illustrated camera system 100 comprises four fixedcameras of which two fixed cameras 102 a, 102 b are visible in FIG. 1.

The fixed cameras 102 a, 102 b are fixed in the sense that they arecapturing a fixed view of the scene, i.e., the same view all the time.Each of the fixed cameras 102 a, 102 b is thus directed in a fixeddirection all the time, and is not arranged to be redirected indifferent directions during use. The fixed cameras may be arranged on arail or guide such that their field of view may be altered manually bymoving the camera along the rail or guide, and by adjusting a tiltangle. The one or more fixed cameras 102 a, 102 b are typically directedin different directions so as to capture images covering differentportions of the scene. The one or more fixed cameras 102 a, 102 b thushave different fields of view, although the fields of view of thecameras 102 a, 102 b may overlap. For example, the one or more cameras102 a, 120 b may be arranged such that their combined field of viewcovers a complete 360° field of view of the scene.

The one or more fixed cameras 102 a, 102 b are each arranged to capturean overview image of the scene. For this purpose, the one or more fixedcameras 102 a, 102 b may each be equipped with a wide-angle lensallowing the one or more fixed cameras 102 a, 102 b to capture a wideangular range of the scene. FIG. 2 illustrates four overview images 202a, 202 b, 202 c, 202 d which are captured by the four fixed cameras ofthe camera system 100. The images 202 a, 202 b, 202 c, 202 d depicts ascene, in this case a parking lot, from different angles. For example,the overview image 202 a may be captured by the fixed camera 102 a, andthe overview image 202 b may be captured by the fixed camera 102 b. Ascan be seen from the images, the fields of view of the one or morecameras 102 a, 102 b overlap. The car which is depicted in the middle ofoverview image 202 a is, for instance, also depicted at the bottom rightcorner of overview image 202 b.

The one or more fixed cameras 102 a, 102 b are mounted such that theirpositions in relation to each other are known. Further, the directionsof the one or more fixed cameras 102 a, 102 b in relation to each otherare typically also known. This does, e.g., allow the angle (i.e., thedifference in direction) between two points in the scene to bedetermined from the coordinates of the two points as depicted inoverview images captured by two different ones of the fixed cameras 102a, 102 b.

The movable camera 104 may be a PTZ camera. The movable camera 104 mayhence be redirected by adjusting the pan and tilt settings, asillustrated by the arrows in FIG. 1, such that its field of view isaltered to cover different parts of the scene. In particular, the PTZ ismounted in relation to the one or more fixed cameras 102 a, 102 b suchthat the PTZ is directable to capture images of different portions ofthe complete field of view covered by the one or more fixed cameras 102a, 102 b. The movable camera 104 may hence be directed to capture imageswhich overlap with each of the overview images captured by the one ormore fixed cameras 102 a, 102 b. FIG. 2 illustrates an image 204captured by the movable camera 104. The image 204 overlaps with theoverview image 202 a and depicts the windscreen of the car that is alsodepicted in the middle of overview image 202 a.

The movable camera 104 has typically a narrower field-of view comparedto the fixed cameras 102 a, 102 b. However, the movable camera 104 isadvantageous in that it can be directed towards and zoom in on differentportions, such as interesting objects, in the scene. Further, as can beseen in FIG. 2, due to the wide-angle optics of the fixed cameras 102 a,102 b, the overview images 202 a, 202 b, 202 c, 202 d are subject tobarrel-distortion causing perspectives and proportions in the images tobe distorted. The image 204 captured by the movable camera 204 does notsuffer from such distortions.

The controller 106 may generally be arranged to control the movablecamera 104 so as to control the movable camera 104 to look in differentdirections. The controller 106 may be implemented in software. For thispurpose, it may comprise a processor, such as a microprocessor, adigital signal processor, of a field programmable gate array, and anon-transitory memory, such as a non-volatile memory. The non-transitoryprocessor may store computer code instructions which, when executed bythe processor, causes the controller to carry out any method describedherein. In particular, it may be cause to carry out a method ofcalibrating a direction of the movable camera 104 with respect to thefixed cameras.

The control of the movable camera 104 may be based on user input, suchas input concerning specific positions in the overview images 202 a, 202b, 202 c, 202 d. For example, the controller 106 may be arranged toreceive input from an operator regarding a specific position, i.e.,pixel coordinate, in one of the overview images 202 a, 202 b, 202 c, 202d and, in response thereto, change the direction of the movable camera104 from its current direction such that the field of view cover aportion of the scene depicted in that position in the overview image 202a, 202 b, 202 c, 202 d. For that purpose, the controller 106 may makeuse of a relation which associates different pixel coordinates in theoverview images 202 a, 202 b, 202 c, 202 d with different directions,i.e., pan/tilt settings, of the movable camera 104. Such a relationdepends on the geometry of the camera system 100 and the optics, such aslenses, of the cameras 102 a, 102 b, 104. However, once the geometry andthe optics has been set, the relation may be determined and stored,e.g., in the form of a function or table, in a non-volatile memory ofthe controller 106. Such a relation is typically pre-determined andpre-stored in the controller 106.

The controller 106 typically works with relative changes in directions,i.e., based on the pre-stored relation it redirects the movable camera104 from a first direction corresponding to a first position in anoverview image, to a second direction corresponding to a second positionin an overview image. Accordingly, upon installation of the camerasystem 100, and before the controller 106 can start to use such arelation for controlling the movable camera 104, the movable camera 104and more specifically its direction needs to be calibrated with respectto the one or more fixed cameras 102 a, 102 b. This means that thecontroller 106 needs to find a correspondence between the initialdirection of the PTZ camera and a position in an overview image capturedby one of the fixed cameras. A method for performing such a calibrationwill now be described with reference to FIGS. 1, 2, 4 and the flowchartof FIG. 3.

In step S02, the controller 106 receives an overview image. The overviewimage is captured by one of the fixed cameras 102 a, 102 b. FIG. 4illustrates an overview image 402 which, for instance may correspond tooverview image 202 a of FIG. 2. In some embodiments, the controller 106receives overview images from more than one or from all of the fixedcameras 102 a, 102 b as shown in FIG. 2.

In step S04, the controller 106 directs the movable camera 104 to lookin a first direction. The first direction is arbitrary and maycorrespond to the direction that the movable camera 104 has uponinstallation.

With the movable camera 104 directed in the first direction, thecontroller 106, in step S06 a receives a first image 404 from themovable camera 104. The first image 404 may, for instance, correspond tothe image 204 of FIG. 2. A field of view 406 of the first image 404partly overlaps a field of view of the overview image 402. In case ofthere being several overview images captured by different fixed cameras,the first image 404 will overlap with at least one of the overviewimages.

Next, the controller 106, in step S06 b, identifies a first set offeatures 408 in the first image 404. The features may, for instance, beedges or corners in the first image 404. The features may further haveassociated attributes, such as size, color and/or direction. Thefeatures may be extracted from the first image 404 by using conventionaltechniques, such as applying filters to the first image 404. An exampleof an algorithm that can be used to extract the features is thescale-invariant feature transform (SIFT). The features are illustratedby “x” in FIG. 4. The controller 106 may also log, i.e., store, thepositions, such as pixel coordinates, of the identified features 408.The first set of features 408 may also be associated with a singlepositional data, such as the mean value of the positions of theindividual features in the set.

In step S06 c, the controller 106 performs feature matching. In moredetail, the controller 106 localizes the first set of features, or asubset thereof, in the overview image 402. In this way, the controller106 may associate the first set of features in the image 404 with asecond set of features 410 in the overview image 402. Step S06 c may becarried out by first extracting features from the overview image 402 andthen matching the first set of features 408 to the extracted features tofind a best match. The best match is then the second set of features410. This may be carried out by using any known feature matchingalgorithm which is suitable for this purpose. For example, algorithmsfrom the fast library for approximate nearest neighbors (FLANN) may beused to find the best matches. In case of several overview images, thefeature matching may be carried out with respect to each of the overviewimages, so as to find the best match among possible matches in alloverview images.

In step S06 d, the controller 106 logs i.e., stores, the positions, suchas pixel coordinates, of the features 410. The second set of features410 may also be associated with a single positional data, such as themean value of the positions of the individual features in the set.

In step S06 e, the controller 106 uses the first set of features 408 andthe second set of features 410 to define a mapping between the image 404captured by the movable camera 104 and the overview image 402 capturedby the fixed camera 102 a, 102 b. The mapping may for example be definedin terms of a transformation matrix which maps points in the image 404to points in the overview image 402. The transformation matrix may,e.g., be determined based on the first set of features 408 and thesecond set of features 410 by applying a least-squares method. Morespecifically, the transformation matrix may be determined as the matrixthat maps the positions of the features 408 of the first set as close aspossible, in a least-square sense, to the positions of the features inthe second set 410.

The controller 106 then uses the defined mapping to map an object in theimage 404 to the overview image 402. The object may for instancecorrespond to an object depicted in the overview image 402, such as thewind screen of the car in shown in image 204 of FIG. 2. In this example,the object does however correspond to the rectangular periphery 412 ofthe image 404. The object may be mapped to the overview image 402 byapplying the transformation matrix to all points that belong to theobject. Alternatively, the object may be represented by a selectednumber of points which are mapped to the overview image by applicationof the transformation matrix. The mapped object 414, i.e. therectangular periphery 412 when mapped to the overview image isillustrated as a deformed rectangle in FIG. 4

Due to different factors, the appearance of the mapped object 414 maylook different than the original object 412. According to a firstfactor, the first set of features 408 may not properly match the secondset of features 410, which results in that the defined mapping does notmap positions in the image 404 correctly to corresponding position inthe overview image 402. According to a second factor, the barreldistortion of the overview image 402 will cause any object in theoverview image 402 to have a different appearance than in the image 404.The second factor will be more pronounced closer to the boundary of theoverview image 402.

The controller 106 may calculate a quality of the mapping based on theappearance of the mapped object 414. The quality of the mapping istypically a measure of how well the mapping preserves the appearance ofan object, and it may be determined to take the first factor or both thefirst and second factor described above into account. Generally, thequality of the mapping may be any metric which is evaluated based on theappearance of the mapped object. This may include evaluating the sizeand/or geometrical shape of the mapped object 414.

In some embodiments, the quality of the mapping is calculated bycomparing the appearance of the object 412 before mapping and the object414 after mapping. In particular, the quality of the mapping may becalculated based on the similarity between the appearance of the object412 before mapping and the object 414 after mapping. This may concernmeasuring the similarity in shape and/or similarity in size. The socalculated quality of the mapping will take both the first factor andthe second factor referred to above into account. In order to measurethe similarity in shape of the object 412, the object may be representedby a plurality of points, such as points on its periphery. For arectangular object, the corner points would preferably be chosen. Onemay then compare the relation between these points before and aftermapping of the object. For example, one may look at how much thedistances between the points have been changed by the mapping. Theamount of change, in relation to the original distance between thepoints may be used as a measure of the similarity in shape. In order tomeasure similarity in size, the area of the object after and beforemapping may be compared.

For a rectangular object, a high similarity in shape is received if thefour corners of the object still define a rectangle after the mapping.On the contrary, a low similarity is received if the four corners aremapped to lie on a line. Further, a high similarity in size is achievedif the size of the rectangle, after mapping, i.e., the area covered bythe object after mapping, has a size which corresponds to the expectedone taking the different lenses of the cameras into account. Incontrast, a low similarity in size is achieved if the size of therectangle after mapping has a much larger or much lower size thanexpected.

In other embodiments, it is desirable that the quality of the mappingonly reflects the first factor mentioned above. In such cases, theeffect of the second factor, i.e., distortions due to barreldistortions, should be removed prior to calculating the quality of themapping. The effect of the barrel distortion may be removed based onknown properties of the lens systems, i.e., the wide-angle lenses, ofthe fixed camera 102 a, 102 b. More specifically, having such lenssystem properties at hand, the controller 109 may calculate what theobject 412 in the image 404 would look like in the overview image 402,i.e., how the wide-angle lens would depict such an object at theposition of the second feature set in the overview image. The controller106 may then proceed to adjust the appearance of the object 412 to whichthe mapped object 414 is to be compared, such that the adjustedappearance is the same as the calculated appearance that the object 412would have when depicted by the wide-angle lens. Having adjusted theappearance of the object 412, the controller 106 may proceed to measurethe similarity of the mapped object 414 and the adjusted appearance ofthe object 412 in accordance with what was described above.

In step S10, the controller 106 performs a first calibration of themovable camera 104. In more detail, the controller 106 correlates thefirst, current, direction of the camera with the positional data of thesecond set of features which was logged in step S06 d described above.In this way, the controller thus finds a first correspondence between adirection of the movable camera 104 and a position in the overview image402.

In step S12, the controller 106 checks whether the determined quality ofthe mapping is below a first threshold T1. The first threshold may be aconstant, predefined, value. In some embodiments, typically when thequality of the mapping has been calculated without removing the effectof the barrel distortion, the first threshold may be a decreasingfunction of the distance of the position of the second set of feature toa center of the overview image. In other words, the quality of themapping may be allowed to be lower closer to the boundary of theoverview image 402 where the effect of the barrel distortion is morepronounced. This may be used as an alternative to correcting for theeffect of the barrel distortion when calculating the quality of themapping.

If the quality of the mapping is not below the first threshold T1 themethod ends, and the first calibration becomes the final calibration.If, however, the quality of the mapping is below the first threshold T1,the controller 106 proceeds to step S14 where it controls the movablecamera 104 to change direction from the first direction to a seconddirection.

As further described above, the quality of the mapping is highlydependent on finding a good match between features in the image 404 andthe overview image 402. A poor match, resulting in a mapping of poorquality, may be due to the movable camera 104 being directed towards aportion of the scene where there are few objects to depict since thisresults in there being few features to extract from the image 402. Thiscould, e.g., happen if the movable camera 104 is directed towards theasphalt of the parking lot depicted in FIG. 2, such as towards thecenter of the overview image 202 d. According to embodiments, themovable camera 104 may therefore be redirected towards a portion of thescene where there are more objects and interesting features. Thatportion of the scene may be depicted in the same overview image 402, orin an overview image captured by another one of the fixed cameras 102 a,102 b. In order to achieve this, the controller 106 may proceed toidentify an area 406 in the overview image 402 (or in one of the otheroverview image captured by another fixed camera) where there are manyfeatures, e.g., where the number of features per unit area exceeds asecond threshold. By using the pre-defined relation described abovewhich associates directions of the movable camera 104 and positions inthe overview image 402, and using the correspondence established by thefirst calibration as an initial calibration of the direction, thecontroller 106 may calculate how to redirect the movable camera 104 fromthe first direction to a second direction in which an image captured bythe movable camera 104 covers the identified area 406 of the overviewimage 402.

According to other embodiments, the influence of the barrel-distortionor other geometric distortions, such as pincushion distortion andmoustache distortion, on the quality of the mapping may instead bereduced by redirecting the movable camera 104 from the first directionto a second direction in which an image captured by the movable camera104 covers a center of the overview image 402, where thebarrel-distortion is less pronounced. Again this may be achieved byusing the pre-defined relation which associates directions of themovable camera 104 and positions in the overview image 402, and usingthe correspondence established by the first calibration as an initialcalibration of the direction.

According to yet other embodiments, the controller 106 may select thesecond direction at random.

Once the movable camera 104 has been redirected, it proceeds to stepS16. In step S16, the controller 106 repeats at least steps S06 a-S06 ddescribed above, but now with the movable camera 104 directed in thesecond direction.

In some embodiments, the controller 106 proceeds to step S20 describedbelow of performing a further calibration once steps S06 a-d have beenrepeated for the second direction.

In other embodiments, the controller 106 also repeats step S06 e withthe camera in the second direction. In such embodiments, the controller106 typically also performs step S18 of comparing the quality of themapping calculated with the camera in the second direction to the firstthreshold. If the quality is good enough, i.e., greater than or equal tothe first threshold, the controller 106 proceeds to step S20 ofperforming a further calibration. If the quality of the mapping is notgood enough, i.e., below the first threshold, the controller 106 onceagain goes back to repeat steps S14 of redirecting the movable camera104, S16 of repeating steps S06 a-e, and S18 of checking whether thequality of the mapping is below the first threshold. The controller 106may keep repeating steps S14, S16, and S18 until the quality of themapping is greater than or equal to the first threshold, whereby thecontroller 106 proceeds to step S20 of performing a further calibration.

In step S20 the controller 106 performs a further calibration, i.e., asecond calibration. In more detail, the controller 106 correlates thecurrent direction of the camera with the positional data of the secondset of features which was logged the last time step S06 d was repeatedunder step S16 as described above. In this way, the controller thusfinds a further correspondence between a direction of the movable camera104 and a position in the overview image 402. Once the controller 106has performed the further calibration, the method ends, whereby thefinal calibration is equal to the further calibration of step S20.

It is to be noted that when there are several fixed cameras 102 a, 102 bhaving a known position and direction in relation to a first one of thefixed cameras, such as in the camera system 100, the movable camera 104may conveniently be calibrated with respect to the other fixed camerasonce it has been calibrated with the first fixed camera. In more detail,once a correlation between a direction of the movable camera 104 and aposition in an overview image captured by a first of the fixed camerashas been established, a correlation between a direction of the movablecamera 104 and a position in an overview image of each of the other ofthe fixed cameras may be calculated. The calculation may be based on thecalibration of the first fixed camera and the known position anddirection of the other cameras in relation to the first fixed camera.

If there are several fixed cameras 102 a, 102 b for which the relativepositions and directions are not known, the movable camera 104 needs tobe calibrated with respect to each of the cameras by applying the methodillustrated in FIG. 3 and described above. In this situation, theproblem may arise that it may be difficult to find a direction and zoomlevel of the PTZ camera which causes it to overlap with a field of viewof a certain one of the fixed cameras. This may for example be the caseif the certain fixed camera has a high zoom level. This problem may besimplified once the PTZ camera has been calibrated with respect to oneor more of the fixed cameras since the fields of view of the alreadycalibrated fixed cameras may be excluded from the search range of thePTZ camera when searching for the field of view of the certain fixedcamera. Also, if several ones of the fixed cameras already have beencalibrated, such as a first, second, and fourth of the fixed cameras ofthe camera system of FIG. 1, the PTZ camera may search for the field ofview of the third camera in between the fields of view of the second andthe fourth camera.

It will be appreciated that a person skilled in the art can modify theabove-described embodiments in many ways and still use the advantages ofthe invention as shown in the embodiments above. Thus, the teachingsshould not be limited to the shown embodiments but should only bedefined by the appended claims Additionally, as the skilled personunderstands, the shown embodiments may be combined.

What is claimed is:
 1. A method of calibrating a direction of a pan,tilt, zoom, PTZ, camera with respect to a first, fixed, camera,comprising: receiving an overview image of a scene captured by a first,fixed, camera, directing a PTZ camera in a first direction, when the PTZcamera is in the first direction, performing the steps of: a) receivingan image of the scene captured by the PTZ camera, wherein a field ofview of the image captured by the PTZ camera partly overlaps a field ofview the overview image, b) identifying a first set of features in theimage of the scene captured by the PTZ camera, c) localizing at least aportion of the first set of features in the overview image so as toassociate the at least a portion of first set of features in the imagecaptured by the PTZ camera with a second set of features in the overviewimage, d) logging positional data of the second set of features in theoverview image, e) defining a mapping between the image captured by thePTZ camera and the overview image based on the at least a portion offirst set of features and the second set of features, and calculating aquality of the mapping, performing a first calibration of the PTZ cameraby correlating the first direction of the PTZ camera with the positionaldata of the second set of features being logged when the PTZ camera isdirected in the first direction, in case the quality of the mapping isbelow a first threshold: redirecting the PTZ camera to a seconddirection, performing steps a)-d) when the PTZ camera is in the seconddirection, and performing a second calibration of the PTZ camera bycorrelating the second direction of the PTZ camera with positional dataof the second set of features being logged when the PTZ camera isdirected in the second direction, wherein step e) further comprisesmapping an object in the image captured by the PTZ camera to theoverview image by using the defined mapping, wherein calculating thequality of the mapping includes calculating a similarity between anappearance of the object in the image captured by the PTZ camera and anappearance of the object after mapping to the overview image.
 2. Themethod of claim 1, further comprising performing step e) when the PTZcamera is in the second direction, wherein the step of performing asecond calibration of the PTZ camera is made on a condition that thequality of the mapping calculated in step e) when the PTZ camera is inthe second direction is greater than or equal to the first threshold. 3.The method of claim 1, further comprising: keep redirecting the PTZcamera to further directions, and repeating steps a)-e) until thequality of the mapping calculated in step e) is greater than or equal tothe first threshold.
 4. The method of claim 1, further comprising:identifying an area in the overview image where a density of features inthe overview image exceeds a second threshold, and selecting the seconddirection on basis of the first calibration of the PTZ camera such thatan image captured by the PTZ camera when directed in the seconddirection covers the identified area in the overview image.
 5. Themethod of claim 1, further comprising: selecting the second direction onbasis of the first calibration of the PTZ camera such that an imagecaptured by the PTZ camera when directed in the second direction coversa center of the overview image.
 6. The method of claim 1, wherein, priorto calculating the similarity, the appearance of the object in the imagecaptured by the PTZ camera is adjusted on basis of properties of a lenssystem of the first camera.
 7. The method of claim 1, wherein the firstthreshold depends on the positional data of the second set of features,such that the first threshold decreases with a distance from a center ofthe overview image.
 8. The method of claim 1, wherein the appearance ofan object is at least one of a size of the object and a geometricalshape of the object.
 9. The method of claim 1, wherein the object in theimage captured by the PTZ camera corresponds to a periphery of the imagecaptured by the PTZ camera.
 10. The method of claim 1, wherein a featureincludes at least one of an edge or a corner in a captured image of thescene.
 11. The method of claim 1, further comprising capturing a furtheroverview image of the scene using a second, fixed, camera having a knownposition and direction in relation to the first camera, and calibratingthe PTZ camera with respect to the further overview image of the scenecaptured by the second camera on basis of the first or secondcalibration of the first camera and the known position and direction ofthe second camera in relation to the first camera.
 12. A system,comprising: a first, fixed, camera arranged to capture an overview imageof a scene, a pan, tilt, zoom, PTZ, camera which is separate from thefirst, fixed camera, and a controller operatively connected to the firstcamera and the PTZ camera, the controller being configured to direct PTZcamera in a first direction, and, when the PTZ camera is in the firstdirection, to perform the steps of: a) controlling the PTZ camera tocapture an image of the scene, wherein a field of view of the imagecaptured by the PTZ camera partly overlaps a field of view the overviewimage, b) identifying a first set of features in the image of the scenecaptured by the PTZ camera, c) localizing at least a subset of the firstset of features in the overview image of the scene so as to associatethe at least a subset of the first set of features in the image capturedby the PTZ camera with a second set of features in the overview image,d) logging positional data of the second set of features in the overviewimage, e) defining a mapping between the image captured by the PTZcamera and the overview image based on the at least a subset of thefirst set of features and the second set of features, and calculating aquality of the mapping, the controller further being configured to:perform a first calibration of the PTZ camera by correlating the firstdirection of the PTZ camera with the positional data of the second setof features being logged when the PTZ camera is directed in the firstdirection, in case the quality of the mapping is below a firstthreshold: redirect the PTZ camera to a second direction, perform stepsa)-d) when the PTZ camera is in the second direction, and perform asecond calibration of the PTZ camera by correlating the second directionof the PTZ camera with positional data of the second set of featuresbeing logged when the PTZ camera is directed in the second direction,wherein step e) further comprises mapping an object in the imagecaptured by the PTZ camera to the overview image by using the definedmapping, wherein calculating the quality of the mapping includescalculating a similarity between an appearance of the object in theimage captured by the PTZ camera and an appearance of the object aftermapping to the overview image.
 13. The system of claim 12, furthercomprising: at least one further fixed camera, wherein the first cameraand the at least one further fixed camera are directed in differentdirections so as to capture overview images covering different portionsof the scene, wherein the PTZ camera is mounted in relation to the firstcamera and the at least one further fixed camera such that the PTZcamera is directable to capture images which overlap an overview imagecaptured by the first camera, and to capture images which overlap anoverview image captured by the at least one further fixed camera.
 14. Anon-transitory computer-readable medium having computer codeinstructions stored thereon for carrying out the method according toclaim 1.